Eecient Algorithms for Decision Tree Cross-validation (extended Abstract)

Extended abstract Cross-validation is a generally applicable and very useful technique for many tasks often encountered in machine learning, such as accuracy estimation, feature selection or parameter tuning. A common property of these tasks is that one wants to validate a learned theory on a set of examples not used for its construction (i.e., an \independent test set"). When insuucient data are available to reliably train on one subset of the data and validate the results on a disjoint subset, cross-validation provides an outcome. It consists of partitioning a data set D into n subsets D i and then running a given algorithm n times, each time using a diierent training set D ? D i and validating the results on D i. The results on each D i are averaged to provide a reliable estimate of the induced model's performance on unseen cases. An often mentioned disadvantage of cross-validation is its computational cost: the learning algorithm needs to be run n times. However, while conceptually this is true, it need not be implemented this way. The purpose of this paper is to show that, in a number of cases, a full cross-validation can be performed with only little overhead over the original induction algorithm. The main contributions of this work are as follows. We show how to extend classical algorithms for decision tree induction 5, 4] in such a way that a full cross-validation is integrated with the induction process at a minimal cost; the key is to observe that in a cross-validation a lot of redundant computations are performed, and by rearranging these computations we can often reuse results instead of recomputing them. We analyse the computational complexity of the novel algorithm, identifying those parameters that innuence the overhead most. It turns out that compared to the standard implementation of cross-validation, our method